This guideline describes the phasing and activities in the following TMap processes:

• Master test plan, managing the total test process
• Development tests
• Acceptance and system tests

Master test plan and other TMap processes

When the test manager, after consultation with the receiving parties, decides what will be tested for each test level, chances are that in the total picture of testing, certain matters will be tested twice unnecessarily. Or that certain aspects are ignored. The method should therefore be vice versa. A test manager, in consultation with the client and other stakeholders, makes a total overview of the distribution across test levels as to what must be tested when and with what thoroughness. The aim is to detect the most important defects as early and economically as possible. This agreement is defined in the socalled master test plan (MTP). This plan constitutes the basis for the test plans for the separate test levels. In addition to this content-based alignment, other types of alignment are: ensuring uniformity in processes (e.g. the defect procedure and testware management), availability and management of the test environment and tools, and optimal division of resources (both people and means) across the test levels.

This means that in addition to test levels like acceptance, system and development tests, the master test plan also plays an important part in TMap. Both for the master test plan and the test levels, it is important to set up a good process for creating plans and preparing, executing and managing activities.

While the goals of the acceptance and system tests differ, these test levels are not described separately, but as one single process. This was decided because the activities in both test levels are virtually the same and separate process descriptions would therefore have (too) much overlap.

In addition to these processes, the process “Supporting processes” has been defined because it is more efficient to organise certain aspects/support centrally than per project. This involves supporting processes for the following subjects:

• Test policy
• Permanent test organisation
• Test environments
• Test tools
• Test professional.

PROCESS : MASTER TEST PLAN, MANAGING THE TOTAL TEST PROCESS

The master test plan provides insight into the various test and evaluation levels to be used, in such a way that the total test process is optimised. It is a management tool for the underlying test levels.

The process “Master test plan, managing the total test process” is split up into two phases: the Planning phase of the total test process and the Control phase of the total test process.

Planning phase of the total test process

The author of the MTP, often the test manager formulates the assignment, taking into account the four BDTM aspects of result, risks, time and cost, in consultation with the client. The test manager then works on the upcoming programme by having discussions with stakeholders and consulting information sources, such as documentation. In parallel, the test manager further elaborates the assignment and determines its scope in consultation with the client. In this phase, the first four steps of BDTM are executed: performing a PRA, establishing a test strategy, estimate and planning (see Figure 1: BDTM steps).


Figure 1: BDTM steps

Further activities in the creation of the plan are: the test manager defines the products that must be delivered by the test levels and makes a proposal as to the setup of the test organisation, centrally and overall per test level. The test manager aligns the infrastructure requirements of the test levels in order to deploy the – often scarce – test infrastructure as efficiently as possible. Test management can also be set up in part at the master test plan level. This can be achieved both by defining central procedures and standards for management and by the central management of certain aspects. Both options aim to prevent reinventing the wheel in the various test levels. The main risks threatening the test process are listed, and possible measures are proposed to manage these risks. As his last step, the test manager submits the master test plan to the client for approval.

Control phase of the total test process

The aim of this activity is controlling the test process, infrastructure and test products at the overall level to provide continuous insight into the progress and quality of the total test process and the quality of the test object. Conformable to the frequency and form defined in the test plan, reports are made on the quality of the test object and the progress and quality of the test process. From the very first test activities, the testers develop a view of that quality. It is important that this is reported in every stage of the test process. The client receives periodical reports, and ad-hoc reports on request, on the condition of the system. Such  reporting and adjustment are a vital part of the BDTM approach (BDTM step 6) and take place at both the level of the master test plan and that of the test level (figure 2 - Control and test processes).



Figure 2 - Control and test processes

PROCESS: DEVELOPMENT TESTS

Development testing is understood to mean testing using knowledge of the technical implementation of the system. This starts with testing the first/smallest parts of the system: routines, units, programs, modules, objects, etc. After it has been established that the most elementary parts of the system are of acceptable quality, the larger parts of the system are subjected to integral testing. The emphasis here is on data throughput and the interfacing between e.g. the units up to the subsystem level.

Place of development tests

The development tests are an integral part of the development work executed by the developer. They are not organised as an autonomous process for an independent team. Despite that, a number of different activities for the development test process, with their mutual order and dependencies, can be identified and described with the aid of the TMap life cycle model. The detailed elaboration may vary per project or organisation and depends, among other things, on the development method used and the availability of certain quality measures.

An important quality measure is the concept of the agreed quality. To this end, the expectations of the client in relation to the craftsmanship and product quality must be made explicit during the planning to set up development testing. Examples of other quality measures are: test driven development, pair programming, code review, continuous integration, and the application integrator approach.

PROCESS: ACCEPTANCE AND SYSTEM TESTS

The acceptance test and system test are considered as autonomous processes to be organised. They have their own test plan, their own budget, and often their own test environment to. They are processes running parallel to the development process, which must be started while the functional specifications are created. The TMap life cycle model is used both in the creation of the test plan and in the execution of the other activities in the test process.

Life cycle model

Like a system development process, a test process consists of a number of different activities. A test life cycle model is necessary to structure the various activities and their mutual order and dependencies. The life cycle model is a generic model. It can be applied to all test levels and test types and used in parallel with the life cycle models for system development. In the TMap life cycle model, the test activities are divided across seven phases: Planning, Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion (see figure 3 - TMap life cycle model). Each phase is split up into a number of activities.

Using a test life cycle model enables the organisation to keep an overview during the test process. By recording what has to be done when, how, with what, where, by whom, etc the claims to and the relationships with other aspects like techniques, infrastructure and organisation are made automatically.


Figure 3 - TMap lifecycle model

The critical path and the shape of the life cycle model

If we were to compare the test process with an iceberg, only the Execution phase would be ‘visible’. This means that only the Execution phase should be on the ‘critical path’ of a project. All activities in the other phases can be done either before or after. The form of the life cycle model (parallelogram) shows that the test phases do not have to be executed strictly sequentially.

Test life cycle model relationships

The relationship between the TMap test life cycle and system development life cycle depends on the system development method used and the relevant test level. However, two ‘fixed’ relationships can be indicated. The start of the Preparation phase has a relationship with the moment at which the test basis becomes available; the start of the Execution phase has a relationship with the moment at which the test object becomes available.

Planning phase

The activities to be executed in the Planning phase create the basis for a manageable and high-quality test process. It is therefore important to start this phase as quickly as possible. The planning phase is an important test phase but is almost always underestimated. Often, the framework for a certain test level is are already defined at the overall level in a master test plan. In this case, the detailed elaboration occurs in this phase.

After the test assignment has been finalised, an overall introduction to the test basis, subject matter and organisation (of the project) is made. It is impossible to test the system completely. Most organisations do not have the time and money for that. This is why the test strategy, estimate and planning are determined according to a risk analysis process (BDTM steps 1 through 4), of course always in consultation with the client. It is then determined which test techniques must be used (BDTM step 5). The objective is to realise the best achievable coverage at the right place within the defined BDTM frameworks. The first steps in setting up the test organisation and test infrastructure are also made. These activities are executed and laid down in the test plan for the relevant test level at the beginning of the test process.

Control phase

The primary test process is rarely executed according to plan. As such, the execution of the test plan also has to be monitored and adjusted, if necessary. This is done in the Control phase. The aim of the activities in this phase is to control and report on the test process in an optimal manner, such that the client has adequate insight into and control over the progress and quality of the test process and quality of the test object.

The test manager and/or administrator manage the test process, infrastructure and test products. Based on these data, the test manager analyses possible trends. He also ensures that he keeps well informed of the developments beyond testing, such as delays in development, upcoming big change proposals, and project adjustment. If necessary, the test manager proposes specific control measures to the client.

Information is the main product of testing. To this end, the test manager creates different kinds of reports for the various target groups, taking account of the BDTM aspects of result, risks, time and cost (BDTM step 6).

Setting up and maintaining infrastructure phase

The Setting up and maintaining infrastructure phase aims to care for the required test infrastructure and resources. A distinction is made between test environments, test tools and workplaces.

Setting up and maintaining the infrastructure represents a specific expertise. Testers generally have limited knowledge in this respect, but are highly dependent on it. No test can be executed without an infrastructure. All responsibilities in relation to setting up and maintaining infrastructure are therefore usually assigned to a separate management department. In a testing programme, therefore, the team will have to collaborate closely with these other parties that may be external to the organisation. This means that test managers are in a situation in which they do not have control over the setup and maintenance of the infrastructure, but depend on it. This makes the setup and maintenance of the infrastructure an important area of concern for the test manager. It is a separate phase in the TMap life cycle model to maintain focus on it during the test. This phase runs in parallel to the Preparation, Specification, Execution and Completion phases. Dependencies with activities in other TMap test phases exist for some Setting up and maintaining infrastructure activities.

Preparation phase

The testability review of the test basis is done in the Preparation phase. The ultimate goal of this phase is to have access to a test basis of adequate quality to design the tests, which has been agreed with the client of the test.

Furthermore an early testability review of the test basis improves quality and prevents potential costly mistakes. This is because the development team works on developing the new information system on the basis of system documentation (which is part of the test basis). This documentation may contain errors that can cause a lot of – often costly – correction work if they are not detected in a timely manner. The earlier an error is found in a development process, the easier (and cheaper) it can be repaired.

Specification phase

The Specification phase specifies the required tests and starting situation(s). The aim is to prepare as much as possible so that tests can be executed as quickly as possible when the developers deliver the test object. This phase starts once the testability review of the test basis is completed successfully. The test specification runs in parallel to, and in the shadow of, the realisation of the software.

Execution phase

The aim of the Execution phase is to gain insight into the quality of the test object by executing the agreed tests. The actual execution of the test starts when the test object, or a separately testable part of the test object, is delivered. The test object is first checked for completeness. It is then installed in the test  environment to assess whether it functions as required. This is achieved by executing a first test, the so-called pretest. This is an overall test to examine whether the information system to be tested, in combination with the test infrastructure, has sufficient quality for extensive testing. The central starting point is prepared if this is the case. The test can be executed on the basis of the test scripts created in the Specification phase. In this case, the starting point must be prepared for the test scripts that are to be executed. The test results are verified during execution. The differences between the predicted and actual results are registered, often in the form of a defects report.

Completion phase

The structured test approach of TMap can yield many benefits in the repeatability of the process. It allows products to be reused in subsequent tests if they comply with certain requirements. This may speed up certain activities. Products may be tangible things like test cases or test environments (testware), but also non-tangible things like experience (process evaluation).

When preserving the testware, a selection is made from the often large quantities of testware. Testware means, among other things, the test cases, test scripts and description of the test infrastructure. During the test process, an attempt was made to keep the test cases in line with the test basis and the developed system. If this was not (entirely) successful, the selected test cases are first updated in the Completion phase before the testware is preserved. The advantage of preserving testware this way is that it can be upgraded with limited effort when the system is changed to execute a (regression) test, for instance. There is therefore no need to design a completely new test.

The test process is also evaluated in this phase. The aim is to learn from the experiences gained and to apply these lessons learned in a new test, if any. It also serves as input for the final report, which the test manager creates in the Control phase.